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ABSTRACT 

This report discusses evaluation as required under 
the Department of Education rules and regulations for the National 
workplace Literacy Program (NWLP) . It is intended to help workplace 
literacy programs meet the requirements of the rules and regulations 
governing the NWLP so that the Department of Education can determine 
the value of programs. The discussion of evaluation focuses upon the 
relationship of the evaluation of workplace programs to the original 
criteria that the secretary of Education used to evaluate 
applications for establishing programs in the first place. A table 
presents the criteria used to evaluate proposals, reworded and 
rearranged to emphasize their use in preparing an evaluation report 
of a program once it has been funded and implemented. These criteria 
specify, in outline, what a well-designed and operated workplace 
literacy program would look like. The paper discusses the following 
topics: purpose of the NWLP, the relationship of literacy ability to 
productivity, the relationship of program design and development to 
evaluation, using the criteria in program evaluation, the need for 
data on program effectiveness, measuring the outcomes of learning, 
measuring improvements in productivity, and developing an attitude 
for inquiry. The paper lists 4 references and 10 resources. (KC) 
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Evaluating 
National Workplace Literacy 
Programs 



Typically, whenever the federal government becomes involved in funding educational 
programs, there is a need for the government officials to review programs to determine whether the 
programs they have funded are, in fact, providing useful educational experiences that meet the 
intent of the Congress, as representatives of the public at large. In this case then, it is advantageous 
to go beyond the self-reports of those involved that they are receiving beneficial educational 
services. There is a need for additional evidence of the effectiveness of the program that is less 
subjective. For instance, if a program aims to improve the ability of employees to read their 
job-related materials, then it is not sufficent for evaluation to report that insuuctors and employees 
say they can read their job-related materials better after they have been in the program for a while. 
Rather, some confirming evidence, such as demonstrated improvements in performing job-related 
reading tasks, would be useful. 

Purpose of the Present Report 

For those who request and receive funding for programs under the federal National 
Workplace Literacy Program, the U. S. Department of Education has published rules and 
regulations regarding the evaluation of such programs (The Federal Register, Friday, August 18, 
1989, pp. 34418-34422). Among other things, the regulations require that each application for 
funds under the program include an evaluation plan (see Table 1, column 6). In this case then, 
program operators must satisfy not only themselves and the other participants active in the program 
as to the value of the program, they must also satisfy the Department of Education which must 
report on the value or the programs to Congress. 

This report expands upon the discussion of evaluation given in the Department of Education 
rules and regulations for the National Workplace Literacy Program (NWLP). It is not a "how to do 
it" guide for evaluation. Rather, it is a discussion of evaluation aimed at helping workplace literacy 
programs meet the requirements of the rules and regulations governing the NWLP so that the 
Department of Education can determine the value of programs. Properly carried out, the process of 
evaluation as outlined by the federal regulations should also help program operators more 
effectively design, develop, implement, operate and improve their workplace literacy programs. 

Because good evaluation starts at the beginning, not the end of a program, in this report, the 
discussion of evaluation focuses upon the relationship of the evaluation of workplace programs to 
the original criteria that the Secretary of Education uses to evaluate applications for establishing 
programs i^the first place. Table 1 presents the criteria used to evaluate proposals reworded and 
rearranged here to emphasize their use in preparing an evaluation report of a program once it has 
been funded and implemented/These criteria specify, in broad outline, what a well-designed and 
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Table 1. IlustratioD of How Criteria for Evaluating Proposals for National Workplace 
Literacy Can be Used to Report Evaluations of Programs. 



Need for the Project 
I 



Documents the needs to 
be addressed by the project 



Program Factors 
2 



Demonstrates the active 
commitment of all partners to 
accomplishing project goals. 



Quality of Training 
3 



Provides training through 
an educdonal agency rather than 
a business, unless transferring 
training to a business is necessary 
and reasonable within the frame- 
work of the project 



Focuses on demonstrated needs 
of adults for workplace literacy 
training 



Documents how needs 
will be met 



Documents benefits to 
adult workers and their industries 
that will result from meeting 
those needs. 



Plan of Operation 

4 

Describes roles of each member 
and each site of the partnership. 



Describes activities to be carried 
out by any contractors. 



Describes roles of othtr organizations 
in providing cash, in-kind assistance, 
or other contributions to the project 

Describes the objectives of the 
project and plan to use project 
resources to achieve each objective. 

Establishes measurable objectives 
for the project that are based on the 
project's overall goals. 



Targets adults with inadequate 
skills aimed at new employment 
career advance mentor increased 
productivity. 

Includes support services based 
on cooperative partnerships to 
overcome barriers to participa- 
tion by adult workers. 

Demonstrates a strong 
relationship between the 
skills taught and the literacy 
requirements of actual tabs. 



Experience & Quality of Personnel 
5 

Provides evidence of the applicant's 
experience in providing literacy 
services to working adults. 

Provides evidence of the experience 
and training of the project director 
in project management 

Provides evidence of the experience 
and training of key personnel in 
relation to the project requirements. 

Indicates amount of time each key 
person will devote to project 



Indicates how nondiscriminatory 
employment practices will be 
implemented. 



Delivers instruction in a readily 
accessible environment conducive 
to adult learning. 



Uses individualized educational 
plans developed jointly by 
instructors and adult learners. 



Uses curriculum materials designed 
for adults that reflect the needs of the 

workplace. 



Evaluation Plan & Cost-Effectiveness 
fi 

Provides clear, appropriate methods 
of evaluation that are objective and 
produces data that are quantifiable. 

Identifies expected outcomes of the 
participants and how those outcomes 
will be measured. 

Determines effects of program on job 
retention, performance, and advance- 
ment 

Obtains data that can be used for 
program improvement 

Provides data indicating costs of the 
program in relation to its benefits. 



Source: Federal Register. VoL 54. No. 159. Friday, August 18. 1989. pp. 34419-34420. Note that the wording and ordering 
here it not the tame at in die federal regulations. The latter should be used for preparing proposals. The present ordering is for 
illustrating how the critieria may be used for the evaluation of programs not proposals. 
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operated workplace literacy program would look like. The evaluation, then, indicates how well the 
program operators implemented the design and operational plans that they submitted for funding, 
what outcomes axe being achieved and how the program can be modified to make it more effective. 

In general, the purpose of evaluation for the Department of Education is to permit the 
Department to place a value on the program in providing services and in demonstrating innovative 
and effective practices. That is, it must first decide whether a proposal for a program is likely to 
result in a needed and effective program, and then it must decide whether the program finally 
developed and implemented is providing an educational experience that meets the stated criteria 
outlined in the original proposal and the intent of the Congress when it passed the bill creating the 
NWLP. 

Evaluation in the NWLP is not something that is accomplished at the end of a program 
development and implementation effort to "see if it worked" Rather, evaluation is an integral pan 
of the original design of the program and an ongoing process that can permit decisions about how 
well the program is achieving one or more of the purposes of the NWLP and, where desirable, to 
improve the program and its value to adult learners, other partners in the project, and the society at 
large. 

Purpose of the National Workplace Literacy Program 

Both literacy providers and the Department of Education must evaluate their programs with 
regard to how well they are achieving the purpose of the National Workplace Literacy Program 
(NWLP). Figure 1 outlines the general purpose of the NWLP and illustrates the types of literacy 
and productivity indicators that might be included in a workplace literacy program 

The general purpose of the NWLP is to provide grants or cooperative agreements involving 
exemplary partnerships of business, industry, or labor organizations and educational organizations 
for projects designed to improve the productivity of the workforce through the improvement of 
literacy skills in the workplace by • 

(a) Providing adult literacy and other basic skills services and activities; 

(b) Providing adult secondary education services and activities that may lead to the 
completion of a high school diploma or its equivalent; 

(c) Meeting the literacy needs of adults with limited English proficiency; 

(d) Upgrading or updating basic skills of adult workers in accordance with changes in 
workplace requirements, technology, products, or processes; 

(e) Improving the competency of adult workers in speaking, listening, reasoning, and 
problem solving; or 

(f) Providing educational counseling, transportation, and child care services for adult 
workers during networking hours while the workers participate in the project (Federal Register, 
August 18. 1989, vol. 54. no. 159. p. 34418). 

As illustrated in Figure 1. the NWLP aims to improve the productivity of the workforce by 
improving the literacy of the workforce. This leads to the two primary questions for evaluation: (1 ) 
does the program improve workforce literacy abilities, and (2) do the improved literacy abilities 
lead to improved productivity? 
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To 
Produce 



The Purpose of the 
Notional Workplace Literacy Program 

is: 



improvements in 
Workforce Literacy 



Leading 
to 



Improvements In 
Workforce Productivity 



Key Evaluation Questions: 



Has the program 
improved the literacy 
abilities of the workforce? 



7\ 



32 



Have the Improved literacy 

abilities of the workforce 

improved the productivity 

of the workforce? 
_ 



Literacy Abilities : 

English Lenguege Communication 
Abilities: 

Listening, S pee king 
Reeding, Writing 

Cognitive Processes: 
Reesoning, Problem Solving 

Content Knowledge: 
Mathematics Knovledge 
Workplace K&ovfedge 
( Personnel Rules; etc.) 
Community & Social Knowledge 

Secondary Education Leading to 
High School Diploma or 
Equivalent 



Productivity indicators : 
Those that may be 



(1) directly 
mediated by 
literacu abilities : 



Examples: 
Converses 
better with 
supervisors & 
coworkers; 
Reads & writes 
job materials 
better; Knows 
more about the 
job, workplace 
end community. 



(2) indirectly 
effected by the 
literacy program : 



Examples. 
Performs job 
tasks better; gets 
to work on time 
more; makes use 
of employee 
benefits more 
appropriately; 
contributes more 
to team efforts; 
Jo* retention; Job 
acvtncement. 



Figure 1 . The purpose of the National Workplace Literacy Program is to improve 
the productivity of the workforce through the Improvement of literecy 
skills in the workplace. Key evaluation questions are ( 1 ) have workforce 
literacy skills been improved and (2) has that led to improvements in 
productivityTSeveral examples of literacy abilities" and "productivity 
indicators" are given. 
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The Relationship of Literacy Ability to Productivity. The basic assumption of the 
NWLP is that there is a relationship between various literacy abilities, as illustrated in Figure 1 , 
and job productivity, as indicated by various measures, examples of which are given in Figure 1 . 

Though this may seem straightforward, it is not true that all aspects of productivity are 
directly mediated by literacy ability. For instance, many job tasks do not require the direct 
application of reading or writing abilities. Nor will they necessarily require specialized knowledge 
that requires reading and writing abilities. Many job tasks can be learned by watching others and 
imitating them 

Therefore, in determining the need for a workplace literacy program that emphasizes 
increasing the reading, writing, or other literacy abilities of the workforce, it is important that 
program developers understand the role of literacy ability in relation to various indicators of 
productivity. Otherwise, if there is simply a blanket assumption that increasing literacy ability will 
increase productivity in some unspecified manner, it may not be possible to demonstate that the 
program has, indeed, increased productivity. 

Figure 1 indicates that some productivity indicators may be directly mediated by literacy 
abilities while others may be only indirectly mediated by literacy ability. For example, being able to 
comprehend oral directions that supervisors provide is direcUy mediated by the ability to 
comprehend the English language, if that is the language used by the supervisor. If the directions 
are not understood, then the worker may not know what to do or how to do it In this case, the job 
tasks may net get done, or they may not be correctly performed, even though the tasks, 
themselves, do not require language comprehension. 

In such circumstances, improving English language comprehension skills may lead to 
improved job task performance not because the tasks require language comprehension, but because 
understanding the directions about what to do and how to do it requires language comprehension. 

On the other hand, because the job tasks do not directly involve the comprehension of 
• English language, it is possible that workers may learn what to do and how to do it by watching 
others. In this case, then, increasing English language skills may not lead to improved task 
performance. Therefore, some other indicator of the increase in productivity due to increased 
language ability should be sought 

Generally speaking, unless a direct relationship to some indicator of productivity can be 
demonstrated in the design of the program, the program developer should not promise to improve 
that aspect of job productivity. However, as a pan of the program evaluation, information about 
aspects of productivity that are not known to be directly mediated by literacy ability should be 
obtained because of die possiblity of the indirect influence of increased literacy ability, or simply 
participation in the literacy program, may have on various indicators of productivity. For instance, 
if having access to education programs boosts employee morale, indicators of productivity such as 
attendance, less tardiness, increased cooperativeness (team work) and so forth may improve. 

Relationship of Program Design and Development to Evaluation 

Because the purpose of the NWLP is to increase workforce productivity through the 
improvement of literacy ability, the design of a workplace literacy program should indicate the 
relationship between literacy ability and productivity, and how the jr%»am intends to increase 
productivity through the improvement of some aspect of literacy ability. 
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This relationship of program design and development to evaluation is illustrated in Table 1 in 
columns 1, 2, and 3. Column 1 calls for a needs assessment that focusses on documenting the 
needs of adults for workplace literacy training, how the needs will be met and how meeting those 
needs will benefit the workers and their industries. Column 2 calls for program factors that 
demonstrate a strong relationship between the skills taught and the literacy requirements of actual 
jobs. Then Column 3 makes clear the need to directly address the program to workplace literacy 
requirements by calling for the use of curriculum materials that reflect the needs of the workplace. 

If the design of the program accomplishes the activities of columns 1 ,2, and 3, then the 
program will have gone a long way toward meeting the requirements of Column 6 for the 
identification of expected outcomes, how those outcomes will be measured, and how those 
outcomes are related to job retention, performance and advancement 

Using Table 1 in Program Evaluation 

As illustrated in the proceeding section, Table 1 outlines criteria for a well -designed 
workplace literacy program. Presumably, since these are criteria used to select projects for 
funding, any projects that receive funding have successfully met these criteria, at least to some 
minimally acceptable extent 

The process of evaluation is the process of turning the various declarative statements, such as 
"Focuses on demonstrated needs of adults for workplace literacy training"(column 1) into 
questions, such as "Does the program focus on the needs of adults for workplace literacy training, 
and how is this demonstrated?" 

By following this procedure of transforming declarative into interrogative statements, Table 1 
can be transformed from a list of criteria for evaluating proposals for programs into criteria for 
evaluating programs of workplace literacy. 

Table 2 illustrates how the categories of Table 1 can be used to summarize the results of 
evaluation studies. In Table 2, findings are summarized from a study of the National Workplace 
Literacy Program by Kutner, Sherman, & Webb (1990; source number 1). Additionally, results 
are summarized from a survey of workplace literacy program evaluations by Mikulecky & 
D'Adaroo-Wrin£iein (1990; source number 2). While the placement of the particular findings in the 
Table 1 categories may be arguable in some cases, the point is that Table 2 illustrates that the 
categories of Table 1 may be used to conduct and report evaluations of NWLP projects. 

For instance, note that in Table 1, category 2-Program Factors, calls for the proposal to 
"Demonstrate a strong relationship between the skills taught and the literacy requirements of actual 
jobs." Then, in Table 2, category 2, it is noted that "Study sites typically assess participant literacy 
levels through standardized tests that are typically used for ABE and are not geared for workplace 
literacy." Because standardized tests do not strongly represent "the literacy requirements of actual 
jobs" they were not considered appropriate for assessing participant literacy levels. This 
observation was included in category 2, rather than in category 6-evaluation- because it illustrates 
the difficulty of matching skills taught (and assessed) to the literacy requirements of actual jobs. 

Additional entries in Table 2 suggest the types of findings that professional evaluators have 
reported from their studies of workplace literacy progams. They illustrate, therefore, the kinds of 
activities and problems that others might consider in evaluating workplace literacy programs. 
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Table 2. Comments from Evaluations of Workplace Literacy Programs. 



Program Factors 
2 



Need for the Project 

I 

Supervisors are invofc*d with 
the workplace literacy projects 
at many of the business sites. 
Initial reluctance of supervisors 
at many of these sites to have 
workers attend clasess on company 
time has been eliminated as 
benefits from the project have 
become apparent (1) 



Formal literacy task analyses 
is the exception rather than the 

rule.(l) 

Businesses at the study sites 
are actively involved with 
recruiting participants by 
identifying potential partici- 
pants. (1) 



Although business sites are support* 
ive of the respective workplace literacy 
projects, few indicated a commitment 
to continue the project without either 
federal or other outside funding. (1) 

Study sites typically assess participant 
literacy levels through standardized tests 
that are typically used for ABE and are 
not geared for workplace literacy. (1) 

Educational providers at the study 
sites are directly responsible for all 
instruction-related activities, including 
conducting literacy task analyses, 
assessing the literacy skills of partici- 
pants, developing instnictional materials, 
and hiring and managing instructors. (1) 



Quality of Training 

3 

A mv bar of project components 
may contribute to the absence of 
retention problems: locating 
instructional services at the work 
site, providing participants with 
monetary incentives, offering a 
supportive learning environment, 
and offering support sen-' ts includ- 
in child care, reimbursement for 
transportation, and counselfrg. (1) 

There is substantial variation worn 
site to site in the total number of 
horns available per training cycle.(l) 

When instructors do not share pro- 
gram goals and resources are 
inadequate instructional quality is 
likely to be inadequate. (2) 



Plan of Operation 
4 



Experience & Quality of Personnel 
5 



Evaluation Plan & Cost-Effectiveness 
6 



Business partners at the study With only one exception, educational 
sites are not heavily involved with providers at the study sites do not 



the day-to-day activities of the 
workplace literacy projects. (1) 



Increased demands for classes are 
reported as indicators of program 
success. (2) 



Anecdotal experiences are reported 

as indicators of program success. 
(2) 

Evaluations generally rely on 
anecdotal evidence, including the 
perceptions of instructors, business 
supervisors, and more senior staff. 
(1) 



have prior experience with workplace 
literacy. (1) 



Almost all of the educational providers 
at the study sites have hired instructors 
who possess experience with ABE or 
ESL programs. (1) 

Most educational providers at the 
study sites do not provide training for 
instructors before instructional services 
began. Most, however, do offer in- 
service training for insmtctors and 
volunteers. (1) 



Source*: (1) Kutner. Shenntn & Webb. 1990; (2)Mileulecky A D'Adano-Weinitein. 1990. 



Study sites do not generally conduct 
formal evaluations of their projects. 
(1) 

Learners were often evaluated by 
supervisors in informal reports. (2) 

Program evaluations tend to be in- 
formal with little or no empirical 
data. (2) 



When programs are evaluated, they 
are often assessed mainly through the 
completion of quesoonaires and/or 
surveys by program participants. (2) 

Some programs do test participants 
both before and after completing the 
program. These results are often 
reported only in general terms as 
indicators ofnmgam effectiveness. (2) 



The Need for Data On Program Effectiveness 

Perhaps the most vexing problem in program evaluation is the determination of whether the 
outcomes that are achieved are useful and justify the expenditures of public funds for this activity 
to meet learner needs rather than for something else. One of the reasons this is such a problem is 
that, while this type of decision making is necessary at the federal level, it is not the major concern 
of local workplace literacy programs. In these programs, program administrators and teachers are 
concerned witn meeting the needs of their adult learners and partners. They are less concerned, if 
at all, with meeting the needs of federal funding agencies for information for decision making. 

While obtaining convincing outcome data is difficult because it is not the highest priority for 
workplace literacy teachers and adult learners, the problem is compounded by the fact that 
hundreds of millions of dollars are spent each year on standardized tests and other assessment 
instruments, throughout the education system in the U.S., and yet no one is satisfied that they a*e 
actually obtaining valid information about "true" achievements. This is indicated by the /act that 
today there are several national activities underway to develop new national examinations to obtain 
a more valid indicator of how well the nation is doing in education. 

In the face of such difficulties in satisfying ourselves that we are doing good, bad, or so-so 
with regard to educational achievement across the spectrum of educational services in the nation, it 
is understandable why workplace literacy operators, teachers and adult learners may be reluctant to 
submit to examinations that they feel are intrusive and nonrepresentative of what they are teaching 
and learning. 

Nonetheless, the fact remains that there is a need, at the federal level, for information 
regarding the effectiveness of the learning activities and outcomes that are taking place under the 
categorical funding of the National Workplace Literacy Program. That is why the criteria for 
proposals includes column 6 of Table 1 -Evaluation Plan & Cost-Effectiveness- which includes the 
requirements for methods of evaluation that are "objective" and which indicate how "outcomes will 
be measured." 

These requirements for' "objectivity" and "measurability" of outcomes in evaluation are not 
baseless requirements of the funding agency. As Table 2 indicates, outside e valuators who have 
examined workplace literacy programs nave independently observed that "progam evaluations tend 
to be informal (unstandardized) with little or no empirical (objective) data (quantifiable measures)". 

In fact, the repeated findings by outside evaluators that programs lack "formal" evaluations, 
that they use "informal" reports, depend primarily upon self-report questionaires with no 
substantiating evidence in more "objective" terms of what is reported, and provide "little or no 
empirical data" are among the most salient outcomes of external evaluations of workplace literacy 
programs (and all other programs in adult literacy or ABE for that matter). 

In short, what these evaluators say is needed is convincing evidence that useful learning 
outcomes are being achieved in programs funded by the NWLP and that this new learning results 
in improved productivity in finding, retaining, performing, or advancing in a job in the workplace. 
While various types of ratings (e.g., supervisor ratings of increased productivity; teacher ratings of 
improvement; adult learner ratings of pre-and post-program increases in learning or productivity) 
provide useful indicators of the program's effects on learning and productivity, such ratings are not 
totally convincing. They are not free of the potential for self-deception that may bias ratings. 
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It is the desire to overcome these kinds of subjective judgments that may lead to inaccurate or 
invalid estimates of the outcomes of programs that lead the federal criteria and evaluation experts to 
call for "objective'V'empiricar, "measurable" outcomes of literacy learning and productivity. 

Measuring the Outcomes of Learning. The goal of the NWLP is to increase workforce 
literacy and thereby improve workplace productivity (see Figure 1). Therefore, the primary 
outcome of a workplace literacy program that needs to be measured is the extent to which literacy 
abilities (defined broadly as the set in Figure 1) have been improved. If there are no improvements 
in literacy abilities then there is no need to look for this non-improvement to increase productivity. 1 

The measurement of literacy abilities ought to reflect the content of what is being taught The 
latter, in turn, will have the best chance of being transferred to the job if it consists of the materials 
and content knowledge needed for getting and performing a job. For instance, if workers in a plant 
need to learn to write reports from production team meetings, it would be better to teach writing 
using the writing of team production reports as the vehicle for teaching proper usage of 
punctuation, planning, presenting, and revising a composition, and other aspects of English 
language, than to use fhe writing of fiction or personal accounts of ones life events. 

The only way to know if growth has taken place in literacy abilities is to measure the abilities 
at the outset of the program, and then again later on. Typically, it will be possible to measure both 
the content knowledge that worker's have relevant to some new domain of learning that they wish 
to command, and the types of knowledge and skill that they possess regarding the uses of language 
and literacy in working with knowledge for doing something or learning something. For instance, 
developing job-related reading task tests (JRTT) using the matetials from literacy task analyses 
(The Bottom line, 1988 ) can permit the assessment of how much of the content knowledge in 
some job or work-related domain the worker knows and how well the worker can apply 
information search, comprehension strategies, and study skills to tocaic and learn knowledge that 
is not known. Administering JRTT as pre-and post-tests will permit an assessment of how much 
improvement has occured in workplace reading skills. 

While JRTT can indicate something of the growth of job-related literacy abilities, they do not 
permit comparisons of growth in one program with growth in another program by other workers. 
Yet the Department of Education needs to know how well programs perform relative to one 
another. For this reason, it is necessary to use one or another nationally normed, standardized 
literacy tests as pre- and post-program measures of the generalizability of growth (see Sticht, 1990 
for an extended discussion of standardized testing in the context ot the Department of Education's 
adult basic education program). 

In using such tests, care should be taken to not over estimate the growth that has taken place. 
This may happen if very large it. jreases in test performance are obtained. For instance, if a worker 
makes a two to five year improvement in test scores in a 20 to 100 hour program, the gain should 
be suspected as inflated due to faulty testing circumstances at the pre-test, post-test, or both. For 



1 It should be noted thai some indicators of productivity may increase due to increased morale when a 
company shows employees that it cares e -tough to provide them an educational opportunity. Thus, a workplace 
literacy program may have an effect on productivity even when there is little or no measurable improvement in 
literacy abilities. 
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this reason, frequency distributions of pre- and post-test scores should be reported, not simply 
means or medians. The latter conceal the variablity in the gain scores that evaluators can use to 
judge the extent to which testing artifacts may be influencing test performance. 

Measuring Improvements in Productivity. The ultimate goal of the NWLP is to 
improve the productivity of the workforce through the improvement of worker's literacy abilities. 
For this reason, after providing convincing evidence that improvements have taken place in literacy 
abilities, the workplace literacy provider needs to present convincing evidence that the 
improvements in literacy have lead to improvements in job productivity. If the materials and tasks 
used in the literacy program are direct simulations of tasks involving the use of literacy abilities on 
the job, then the JRTT or other literacy assessments are direct indicators of increased productivity 
in performing the literacy-mediated components of job tasks. 

As Figure 2 indicates, it is important to distinguish those aspects of productivity that can be 
shown to be directly mediated or affected by literacy abilities and those that are capable of being 
affected by factors other than increases in literacy abilities (see footnote on page 9). Workplace 
literacy programs should only be held accountable for improving those aspects of productivity 
directly mediated by literacy abilities. And even then care should be exercised in building 
expectations for the effects of literacy education on productivity. Too many other factors, such as 
poor supervision, bad managment practices, substance abuse, and so forth may influence 
productivity to expect improved literacy to overcome any and all productivity problems. Workplace 
literacy providers should not promise more than they can be certain of delivering when it comes to 
improving productivity. 

One of the most frequently used methods of evaluating changes in productivity is to have 
supervisors provide pre-and post-program ratings of improvements in such factors as attendance, 
lateness for work, accuracy in performing job tasks, reductions in errors or wastage of material, 
compliance with safety rules, or other types of indicators of productivity. While this information is 
useful in evaluating the effects of literacy education on productivity, it is subject to the criticisms of 
subjective ratings given above. In this regard, it is useful to have ratings of literacy program 
participants and non-participants from supervisors who do not know which employees have been 
involved in literacy training. This reduces the likelihood of positive bias for program participants 
on the part of the supervisors. 

If possible, company records of performance appraisals of participants before the literacy 
training and after should be obtained and summarized. Records of waste, returned products, 
customer complaints and other objective indicators of productivity should be sought to support the 
rating information. Additional examples of productivity measuies can be found in the list of 
resources included with this paper. 

Developing an Attitude for Inquiry. As stated at the outset, one of the goals of 
evaluation is to permit the improvement of programs, not to simply decide if they work or not The 
gathering of the types of information discussed in this paper should be undertaken in die spirit of 
inquiry - always questioning, seeking information, ana using that information to modify programs 
to make them more effective. Programs that seek to instill the love of lifelong learning in the 
workforce by starting learners off with the first steps into workplace literacy, should themselves 
exhibit positive attitudes toward learning - learning what they are doing, how they are doing it, and 
what might be done to improve what they are doing. Programs that hope to make critical thinkers 
of others should become models of critical thinking themselves. Good evaluation requires critical 
thinking, continuous learning, and thorough documentation to permit others to properly place a 
high value on good works. 
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